The increase in the volume of blocks of information or "documents" transmitted over computer networks has placed significant demands on those networks, particularly wide area networks such as intranets, the Internet or the World Wide Web. The documents (e.g., an HTML web page and associated files) requested may be quite large and thus require a great deal of transmission bandwidth and a significant amount of time to retrieve the document. In the case of a wide area network, the requesting computer or "client" may be a long distance from the computer or "server" on which the document resides. When large distances are involved, the expense associated with the use of transmission lines for the communications naturally increases. In addition, the delay in the client receiving the document also increases which may result in a failure of a particular application, wasted computer time, and frustration on the part of users.
In order to decrease costs and transmission time, networks have begun to store or "cache" documents at various locations to reduce the distance between a user and the stored document. For instance, if a client in San Francisco requests a document which is located in New York, that request will generally flow through an intermediate server located in San Francisco. That intermediate server will request the document from the New York server and then transmit the document to the client. If the network is configured for caching, the intermediate receiver may also retain a copy of the document. When the intermediate server next receives a request for the document, the request will not be forwarded to the New York server. Instead, the transmission is interrupted by caching software at the intermediate server which retrieves the document from the intermediate server and forwards it to the client. Since all transmissions relating to fulfilling the second request for the document take place in the San Francisco area, transmission time and cost is greatly reduced.
One problem associated with caching documents is the possibility that the document cached at the intermediate server may become stale. The author of the document or others may make changes to the document at the "content providing" server in New York. If such a change is made the cached document in San Francisco is outdated or stale. In order to limit the number of stale documents which are cached, the intermediate server may destroy documents after a certain period of time or may destroy a document if a request for that document has not been received within a certain period of time. Such solutions are undesirable because they do not ensure that a cached document is not stale and require complete recopying of a document once it has been removed or flushed from a cache site. In other systems, the content providing server will notify any caching intermediate servers when a document has been altered or the intermediate server will request assurance that the document has not been changed prior to transmitting its own cached copy of the document. Such systems, however, also generally require complete retransmission of the document, along with the attendant delays, if the document has been changed.
Certain documents are not stored at a content providing server as a complete file, but are instead generated dynamically using a program which acts on data. The data may or may not be stored at the content providing server and may be subject to change or updating. Documents which are dynamically generated are usually done so in order to save storage space and because the data on which the document is based is frequently changed or updated. Caching of such dynamically generated documents is therefore problematic because those documents may require frequent updating and might also require retrieval of data, not just from the content providing server, but from other sources as well.